Evaluations

• Important when developing LLM applications

• Standard for eval in general: https://cookbook.openai.com/examples/evaluation/getting_started_with_openai_evals

• Metrics:

  • RAGAS: https://docs.ragas.io/en/stable/index.html
    • The standard approach/metrics for most RAG-based application evaluation
    • Most eval library/platform/solution will incorporate RAGAS among its metrics

• Platform/Services:

  • https://docs.smith.langchain.com/concepts/evaluation#experiments
  • https://www.braintrustdata.com/